In-Sample and Out-of-Sample Fit: Their Joint Distribution and its Implications for Model Selection and Model Averaging (Criteria-Based Shrinkage for Forecasting)

We consider the case where a parameter, ; is estimated by maximizing a criterion function, Q(X ; ). The estimate is then used to evaluate the criterion function with the same data, X , as well as with an independent data set, Y. The in-sample …t and out-of-sample …t relative to that of 0; the “true”parameter, are given by Tx;x = Q(X ; ̂x) Q(X ; 0) and Ty;x = Q(Y; ̂x) Q(Y; 0). We derive the limit distribution of (Tx;x; Ty;x) for a large class of criterion functions and show that Tx;x and Ty;x are strongly negatively related. The implication is that good in-sample …t translates directly into poor out-of-sample …t. This result forms the basis for a uni…ed framework for discussing aspect of model selection, model averaging, and the e¤ects of data mining. The limit distribution can also be used to motivate a particular form of shrinkage, called qrinkage, where in-sample parameter estimates are modi…ed to o¤-set the over…t of the criterion function, hence the name. This form of shrinkage is particularly simple in the context of regression models, such as the factor-based forecasting models.